High-Performance Storage Support for Scientific Big Data Applications on the Cloud

This work studies the storage subsystem for scientific big data applications to be running on the cloud. Although cloud computing has become one of the most popular paradigms for executing data-intensive applications, the storage subsystem has not been optimized for scientific applications. In particular, many scientific applications were originally developed assuming a tightly-coupled cluster of compute nodes with network-attached storage allowing massively parallel I/O accesses—the high-performance computing (HPC) systems. These applications, in turn, struggle in leveraging cloud platforms whose design goal is fundamentally different than that of HPC systems. We believe that when executing scientific applications in the cloud, a node-local distributed storage architecture is a key approach to overcome the challenges from the storage subsystem. We analyze and evaluate four representative file systems (S3FS, HDFS, Ceph, and FusionFS) on multiple platforms (Kodiak cluster, Amazon EC2) with a variety of benchmarks to explore how well these storage systems can handle metadata-intensive, write-intensive, and read-intensive workloads. Moreover, we elaborate the design and implementation of FusionFS that employs a scalable approach to managing both metadata and data in addition to its unique features on cooperative caching, dynamic compression, GPUaccelerated data redundancy, lightweight provenance, and parallel serialization. Dongfang Zhao University of Washington, Seattle, WA, USA. e-mail: [email protected] Akash Mahakode Illinois Institute of Technology, Chicago, IL USA. e-mail: [email protected] Sandip Lakshminarasaiah Illinois Institute of Technology, Chicago, IL USA. e-mail: [email protected] Ioan Raicu Illinois Institute of Technology, Chicago, IL, USA. e-mail: [email protected]